Papers with AlpacaEval 2
SPO: Self Preference Optimization with Self Regularization (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing reference-free preference optimization methods exhibit higher training efficiency but are prone to overoptimization, leading to performance degradation. |
| Approach: | They propose a reference-free preference optimization method that replaces the logsigmoid loss function with a SiLU function to improve the model's performance. |
| Outcome: | The proposed method achieves 7% improvement over SimPO on AlpacaEval 2 and MT-Bench. |
Ambiguity Awareness Optimization: Towards Semantic Disambiguation for Direct Preference Optimization (2025.emnlp-main)
Copied to clipboard
| Challenge: | Direct Preference Optimization (DPO) is a widely used reinforcement learning from human feedback (RLHF) method across various domains. |
| Approach: | They propose an approach that automatically re-weights ambiguous content to reduce ambiguities by calculating semantic similarity from preference pairs. |
| Outcome: | The proposed approach outperforms state-of-the-art approaches in performance across multiple model scales and widely adopted benchmark datasets. |
Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judge (2025.emnlp-main)
Copied to clipboard
Tianhao Wu, Weizhe Yuan, Olga Golovneva, Jing Xu, Yuandong Tian, Jiantao Jiao, Jason E Weston, Sainbayar Sukhbaatar
| Challenge: | Existing methods for improving large language models have focused on improving model responses rather than judgment capabilities, resulting in rapid saturation during iterative training. |
| Approach: | They propose an iterative Meta-Rewarding step where the model judges its own judgements and uses that feedback to refine its judgment skills. |
| Outcome: | The proposed model improves Llama-3-8B-Instruct from 22.9% to 39.4% on AlpacaEval 2 and 20.6% to 29.1% on Arena-Hard. |
Capturing Nuanced Preferences: Preference-Aligned Distillation for Small Language Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for aligning small language models with human values model preference knowledge from large language models (LLMs) however, this limitation hinders student SLMs from capturing nuanced preferences for multiple responses. |
| Approach: | They propose a framework which models teacher's preference knowledge as a probability distribution over all potential preferences, thereby providing more nuanced supervisory signals. |
| Outcome: | The proposed framework outperforms existing methods on four benchmark tasks and achieves 20% improvement on AlpacaEval 2 and Arena-Hard. |
DiffPO: Diffusion-styled Preference Optimization for Inference Time Alignment of Large Language Models (2025.acl-long)
Copied to clipboard
Ruizhe Chen, Wenhao Chai, Zhifei Yang, Xiaotian Zhang, Ziyang Wang, Tony Quek, Joey Tianyi Zhou, Soujanya Poria, Zuozhu Liu
| Challenge: | Inference-time alignment approaches still face limitations due to policy-specific value functions and latency during the inference phase. |
| Approach: | They propose an efficient and policy-agnostic preference optimization method that avoids time latency associated with token generation. |
| Outcome: | The proposed method achieves a favorable trade-off between alignment quality and inference-time latency. |
Weights-Rotated Preference Optimization for Large Language Models (2025.emnlp-main)
Copied to clipboard
Chenxu Yang, Ruipeng Jia, Mingyu Zheng, Naibin Gu, Zheng Lin, Siyuan Chen, Weichong Yin, Hua Wu, Weiping Wang
| Challenge: | Existing methods to align large language models with high reward hacking are limited by the complexity of the parameter space and the complexity. |
| Approach: | They propose a weights-rotated preference optimization algorithm that constrains the output layer logits with the KL divergence inherited from DPO and fine-tunes the intermediate hidden states. |
| Outcome: | The proposed algorithm achieves a 3.27-point improvement on AlpacaEval 2 and surpasses the best baseline by 6.2 to 7.5 points on MT-Bench with merely 0.015% of the trainable parameters. |
The Best of Both Worlds: Combining Parallel and Sequential Inference Scaling via Aggregation Fine-Tuning (2026.findings-acl)
Copied to clipboard
| Challenge: | Empirical results show that AFT-trained models achieve substantial gains with test-time scaling. |
| Approach: | They introduce a supervised fine-tuning paradigm where models synthesize multiple draft responses into a single, refined answer. |
| Outcome: | Empirical results show that AFT-trained models outperform baseline models while eliminating external guidance. |